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10 TECHNICAL FIELD 

The present invention concerns a method of seeking images, 
using an example image, from amongst a plurality of images stored in a 
database, each of the stored images being associated with a data item 
representing at least one characteristic of the visual content of the image. 

15 The invention also concerns a device able to implement such a 

method. 

BACKGROUND OF THE INVENTION 
A conventional method of seeking digital images in a database 
containing images is based on a system of indexing the images in the 
20 database. 

The purpose of the indexing system is to associate, with each 
image in the database, an item of information characteristic of the content of 
the image referred to as the index of the image. All these information items 
form the index of the database. 
25 A user can then interrogate the image database through a request 

containing an item of information characteristic of the type of image sought. 
The content of the request is then compared in accordance with a search 
strategy with the content of the index of the database. 
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Finally, the image in the database whose indexed information item 
has the greatest similarity to the content of the request is then extracted. A 
plurality of images extracted from the database can also be presented to the 
user, ordered according to their degree of similarity to the request. 
5 In a traditional system of indexing digital images, the index of the 

database is composed of textual descriptors of the images stored. 

The request of the user then consists of key words describing the 
characteristics of the content of the image to be sought. 

This type of indexing by textual descriptors has the drawback of 
10 being imprecise, in particular because the same image may be described in 
different ways by different users. 

In order to mitigate this type of drawback, the need has therefore 
been felt to develop techniques for representing and extracting the semantic 
content of a digital image. 
15 Methods have appeared in which an image is characterised 

according to the distribution of the colours or textures making it up. 

In other methods, an image is characterised by the shape or 
contour of an object making it up. 

However, all these descriptors, referred to as "primitives", of the 
20 image, reflect only physical characteristics of the image, and are therefore of a 
low semantic level. 

In order to increase the semantic character of the indexing of the 
images, indexing systems which use a combination of low-level primitives are 
beginning to appear. 

25 One of the most well-known is certainly the QBIC ("query-by- 

image-contenf) system developed by IBM. 

To obtain more details on this system, reference can be made to 

the article "The QBIC Project: Querying Images by Content Using Color, 

Texture, and Shape", by Niblack, W et al., which appeared in IBM Computer 
30 Sciences Research Report, pp 1-20 (1 st Feb. 1993). Reference can also be 

made to US Patent No 5579471 of IBM entitled "Image query system and 

method'. 
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The QBIC system makes it possible to find digital fixed (or video) 
images from a request by means of the example. In the particular case of fixed 
images, this request is defined either as a complete image or as an object with 
a rectangular or arbitrary shape, extracted from an image or defined by the 
5 user. 

The content of the images in the database is characterised by its 
colour distribution (histogram), texture and shape. 

Where the request is defined as a portion, also referred to as a 
region, of an image, the similarity measurement also takes account of the 
10 spatial position of this region. 

However, the request relates at best only to a single portion of the 
example image. 

Moreover, the user has no possibility of designating particular 
areas of the example image whose content he does not wish to find in the type 
15 of images which he is seeking. 

In addition, the user cannot specify to the search system that he 
wishes for the search to be made only from the content of particular regions 
which he has defined in the example image, without taking into consideration 
the content of the remainder of the image. 
20 These particular areas or regions of the example image are 

referred to, in the remainder of the description, as "regions of interest". 

SUMMARY OF THE INVENTION 
The present invention aims to remedy the aforementioned 
drawbacks of the image search systems of the prior art. 

25 To this end, the present invention concerns a method of seeking 

images, from an example image, from amongst a plurality of images stored in a 
database, each of the stored images being associated with a data item of a first 
type, referred to as the index of the stored image, representing at least one 
characteristic of the visual content of the image. 

30 In accordance with the invention this image seeking method 

comprises the following steps: 
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- receiving a data item of a second type representing the location 
of at least one region of interest in the example image; 

- for each region of interest, receiving a data item of a third type 
indicating a type of taking into account of the content of the said region of 

5 interest for the seeking of images; 

- calculating a data item of a fourth type, referred to as the index 
of the example image, representing at least one characteristic of the visual 
content of the example image, the method of calculating the data item of the 
fourth type depending on the data item of the second type and the data items of 

10 the third type; 

- calculating a similarity between the example image and each of 
the images amongst at least one subset of the stored images, the similarity 
being calculated from said data item of the first type associated with the stored 
image and the data item of the fourth type associated with the example image; 

15 - supplying at least one image, referred to as the result image, in 

the database, this result image or images being selected from amongst the 
stored images in the database according to its or their degree of similarity with 
the example image. 

Thus, on the one hand, by enabling the user to supply an item of 
20 information (a data item of the third type) enabling him to specify the way in 
which he wishes the content of the regions of interest which he has defined in 
the example image to be taken into account in the seeking of images, the 
search method of the present invention offers the user great flexibility in 
defining his request precisely. Moreover, by adapting the calculation of the 
25 index of the example image to these data items of the third type, there is 
obtained a calculation of similarity between the example image and any stored 
image which is very precise and which faithfully reflects the choices of the user 
in the definition of his request. 

According to a preferred embodiment of the invention, the data 
30 item of the third type, denoted V r , associated with a region of interest, denoted 
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ROI r , is a scalar data item which can take all the values lying between a 
predefined lower value, V min , and a predefined higher value, V max . Moreover, if 
this data item of the third type V r is equal to the predefined lower value V mint the 
content of the images sought must not be similar to the content of the 
5 corresponding region of interest ROI r ; if this data item of the third type V r is equal 
to the predefined higher value V max , the content of the images sought must be 
similar to the content of the corresponding region of interest ROI r ; finally, if this 
data item of the third type V r lies strictly between the lower predefined value 
V m in and the higher predefined value V max , the content of the images sought 

10 must be more or less similar to that of the corresponding region of interest ROI r 
depending on whether the value of V r is close to V max or is close to V min , the 
overall content of the example image also having to be taken into consideration. 

In this way, the user has a simple and effective means for defining 
his choice with regard to the search strategy which he wishes to apply according 

1 5 to the regions of interest defined in the example image. Moreover, the choice of 
the user 50 so expressed is easily applicable through the index of the example 
image, whose calculation method depends on the data of the third type. 

According to one characteristic of the preferred embodiment of the 
invention, the data item of the first type constituting the index of a stored image 

20 under consideration consists of a histogram of colours relating to the overall 
content of the image. 

Thus, by the use of the distribution of colours in the stored image 
under consideration as an index of this image, the precision of the index is 
increased compared with the use of another descriptor such as the texture. 

25 According to a particular aspect of the preferred embodiment of the 

invention, if all the data items of the third type are equal to the lower predefined 
value Vmin, or if all the data items of the third type are equal to the higher 
predefined value V max ; or if each of the data items of the third type is either 
equal to V m in or equal to V max , then the step of calculating the index of the 

30 example image includes a step of calculating a vector, (G R (Q)), each 
component of which consists of the histogram of colours representing the visual 
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content of one of the regions of interest, this vector constituting the index of the 
example image. 

Thus, under the conditions mentioned above relating to the data 
of the third type, an index of the example image is obtained which represents 
5 only the content of the regions of interest of the example image, thus applying a 
strategy in accordance with the choice of the user expressed by the values 
which he has allocated to the data of the third type. 

According to another particular aspect of the preferred 
embodiment of the invention, if all the data of the third type are strictly between 
10 the lower predefined value V min and the higher predefined value V max , then the 
step of calculating the index of the example image includes the following steps: 

- calculating a matrix (W) with M rows and M columns, where M is 
an integer number designating the number of colours available, each element 
of whose diagonal corresponds to one of the M colours available, each of the 

15 elements of the diagonal having a value which is calculated as a function of the 
dominant character of the colour associated with this element in the regions of 
interest associated with the example image, and of the data item of the third 
type associated with each of the regions of interest; 

- calculating the histogram of colours (H M (Q)) representing the 
20 overall visual content of the example image (Q); 

- defining the index of the example image (Q) as being the result 
(X(Q)) of the product of the matrix (W) and the histogram of colours (H M (Q)) 
representing the overall visual content of the example image (Q). 

Thus, under the conditions mentioned above relating to the data 
25 of the third type, there is obtained an index of the example image consisting of 
a histogram of colours whose colour elements are weighted, through the 
product of the histogram and the matrix, as a function of the presence in a 
dominant fashion of this colour element in the regions of interest defined in the 
example image. 

30 According to yet another particular aspect of the preferred 

embodiment of the invention, when the data of the third type are not all equal to 
the lower predefined value V min , and are also not all equal to the higher 



predefined value V max , and are also not each equal either to V min or to V max , and 
also not all strictly between V min and V max , the index of the example image 
consists of the result (X(Q)) of the product of the matrix (W) and the histogram 
of colours (H M (Q)) representing the overall visual content of the example image 
5 (Q), and of the vector, (G R (Q)), each component of which consists of the 
histogram of colours representing the visual content of one of the regions of 
interest. 

Thus, again, under the conditions mentioned above, relating to the 
data of the third type, there is obtained an index of the example image whose 
10 calculation allows the adaptation of the image search strategy to the values of 
the data of the third type supplied by the user. 

In practice, the data item of the second type representing the 
location of at least one region of interest in the image consists of a set of two- 
dimensional points indicative of the shape of the region or regions of interest 
1 5 and its or their location in the image plane of the image. 

In this way, there is obtained a mode of representing a region of 
interest which is inexpensive in terms of computer resources used and which is 
sufficient from the point of view of the use which is made thereof. 

Other advantageous characteristics of the image search method 
20 according to the present invention are defined in the claims attached to the 
present description of the invention. 

According to a second aspect, the present invention concerns a 
device for seeking images, from an example image, from amongst a plurality of 
images stored in a database. 
25 In accordance with the invention this device comprises means 

adapted to implement an image search method according to the invention, as 
defined above. 

The present invention also concerns a computer, comprising an 
image search device according to the invention or means adapted to implement 
30 the image search method according to the invention. 

The invention also relates to a computer program including one or 
more sequences of instructions able to implement the image search method 
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according to the invention when the program is loaded and executed in a 
computer. 

The invention also relates to an information carrier, such as a 
diskette or compact disc (CD), characterised in that it contains such a computer 
program. 

The advantages of this device, computer, computer program and 
information carrier are identical to those of the method as succinctly disclosed 
above. 

Other particularities and advantages of the invention will also 
emerge from the following description. 

BRIEF DESCRIPTION OF THE DRAWINGS 

A preferred embodiment of the present invention will now be 
described with the help of the accompanying drawings, in which: 

- Figure 1 is a block diagram illustrating the general architecture of 
an image search device in a database according to the present invention; 

- Figure 2 depicts a flow diagram illustrating the sequence of the 
steps of an image search method according to the present invention; 

- Figure 3, composed of Figures 3a, 3b, 3c, depicts a flow diagram 
illustrating the method of indexing the example image according to a preferred 
embodiment of the invention; 

- Figure 4, composed of Figures 4a, 4b, 4c, depicts a flow diagram 
illustrating the method of calculating the similarity between an example image 
and an image in the database according to a preferred embodiment of the 
invention; 

- Figure 5 depicts schematically a computer adapted to implement 
the image search method according to the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 
A description will first of all be given, with reference to F5gy re 1 , of 
a block diagram illustrating the general architecture of an image search device 
(1) according to the present invention. The device 1 contains hardware 
elements and software elements. 
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The device depicted in Figure 1 comprises an image data 
input/output unit 10 intended to allow the inputting, into a database 11, of new 
images to be stored, and to recover a stored image and/or the data indexed at 
the stored image. 

5 A unit 30 for inputting an example image is associated with an 

input device 31 to enable a user to input or indicate an example image whose 
comparison with the stored images in the database will make it possible to 
obtain one or more images as a result of the search. These images can be 
displayed on a display unit 90. 

10 The device 1 also comprises an image data storage unit 20, 

intended to temporarily store the data recovered from the database 11 or the 
data associated with the example image obtained by the units 30 and 31. 

The device of Figure 1 also comprises a unit 40 for recovering 
index data associated with the images stored in the database 1 1 , or extracting 

15 index data associated with images which are to be stored in the database, 
depending on circumstances. 

In the same way a unit 50 is responsible for extracting or 
recovering the index data of the example image. 

A similarity calculation unit 60 is responsible for evaluating the 

20 similarity of the example image with the database images. An image selection 
unit 70 is then responsible for sorting the database images according to their 
similarity with the example image, and for selecting one or more images stored 
as a result of the search. In a preferred embodiment of the invention the 
selected images are then displayed by the display unit 90. 

25 Finally, the image search device depicted in Figure 1 has a control 

unit 80, for controlling the overall functioning of the device. 

Each image stored in the database 11 is indexed as follows. 
When a new image is to be stored in the database, there is previously extracted 
from it a data item representing at least one characteristic of the visual content 

30 of the image. In accordance with one preferred embodiment of the invention, 
this data item is characteristic of the distribution of the colours (histograms of 
colours) in the image. 
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The invention also applies to the case where the images are 
stored in compressed form. In this case, an image is represented by a bit 
stream. For example, where the compression algorithm uses digital wavelet 
transform known from the state of the art, it would be possible to characterise 
5 the content of the original image by considering a particular sub-band of the 
image. 

Finally, with each image stored in the database there is 
associated a data item, referred to in the remainder of the description as 
"content data item" or "data item of the first type", representing at least one 
10 characteristic of the visual content of the image. This data item is referred to as 
the "index" of the stored image under consideration. 

The indexing of an image which is to be stored in the database is 
effected as mentioned previously, by the stored image index 
extraction/recovery unit 40. 
15 With reference now to Fogure 2 f a description will be given of the 

image search method according to the present invention implemented by the 
image search device (1) depicted in Figure 1. 

The search method commences by step S1 in which a human 
user uses the example image input unit 30 associated with the input device 31 
20 in order to define an example image which will serve as a reference for seeking 
images in the database. The user has the choice, for defining the example 
image, between designating an image in the database, or supplying an image 
external to the database. 

According to the preferred embodiment of the invention, the input 
25 device 31 has a pointing device such as a mouse, as well as a keyboard, and 
the display unit 90 includes a screen. 

To enable the user to choose an image in the database as an 
example image, the search device displays on the screen 90 icons representing 
the images in the database. The user can then select, by means of the mouse, 
30 the icon of the image whose content seems to him to be closest to that of the 
image or images which he is seeking. As a variant, the user can also use the 
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keyboard for entering the reference of the stored image which he chooses as 
an example image. 

If the user chooses to provide an image external to the database, 
he can for example supply to the search device 1 the access path to the 
example image which will be accessible via a disk drive or compact disc (CD 
ROM) drive integrated into the input device 31 . 

Returning to Figure 2, once the user has entered (step S1) an 
example image in the image search device 1 , the latter defines, at step S2, one 
or more regions of interest (ROIs) associated with the example image. 

According to the preferred embodiment of the invention, the 
example image chosen is displayed on the screen, and the user uses the 
mouse of the input device 1 for drawing the contour of a region of interest (ROI) 
of the image, for example using the mouse according to the method known by 
the expression "drag and drop". The contour drawn may be of a normal 
geometric shape such as a circle, or a polygon, or any shape. Once the 
contour of the region of interest has been drawn, the user validates the contour, 
for example by pressing a key on the keyboard. The contour is then 
transformed into a set of 2D points which defines the shape and spatial position 
of this region on the image plane. 

The user has the possibility of defining several regions of interest 
according to the same operation. 

As can be seen in box S2 in Figure 2, this set of data indicating 
regions of interest associated with the example image Q is denoted ROI(Q). 

In accordance with the invention, at the step S3 which follows, the 
user has the possibility of associating, with each region of interest ROI r which 
he has defined, a data item V r whose value indicates the manner according to 
which he wishes for the visual content of the region of interest concerned to be 
taken into account by the image search process. 

According to one characteristic of the chosen and illustrated 
embodiment of the invention, this data item associated with each region of 
interest is a scalar data item, entered either by means of the keyboard or by 
means of a pointing device, a mouse for example. 
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More precisely, in the embodiment of the invention, this scalar 
data item can take all the values lying between a lower limit (denoted V min ) and 
an upper limit (denoted V max ). 

According to one advantageous characteristic of the embodiment 
5 of the invention chosen and illustrated, the meaning given to the value of each 
data item V r associated with a region of interest ROI r is as follows, 
o If V r is equal to V min , this means that the user is seeking images in the 
database whose visual content is not similar to that of the associated region 
of interest ROI r . 

10 o If V r is equal to V max , this means that the user is seeking images whose 
visual content is similar to that of the associated region of interest ROI r . 
o If V r is strictly between V min and V max , this means that the user is seeking 
images whose visual content is similar to that of the example image taken in 
its totality, but more particularly taking account of the content of the 
15 associated region of interest ROI r . More particularly, the closer V r is to V ma x 
the closer the visual content of the images sought must be to that of the 
associated region of interest. Conversely, the closer V r is to V min , the less 
close must be the visual content of the images sought to that of the 
associated region of interest. 
20 In practical terms, in the preferred embodiment of the invention, 

the lower limit V min is equal to zero and the upper limit V ma x is equal to one. 

All the data corresponding to the regions of interest defined in the 
example image Q, that is to say all the 2D points associated with the regions of 
interest, denoted ROI(Q), and the set, denoted V(Q), of the data V r associated 
25 with the regions of interest, are then stored in the image data storage unit 20. 

Returning to Figure 2, at step S4 the index of the example image 
Q is calculated. 

In accordance with the invention, this index consists of a data item 
representing at least one characteristic of the visual content of the example 
30 image, whose calculation mode takes into account the value of the data V r 
associated with the regions of interest. 
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The method of calculating the index of the example image will be 
detailed below in relation to Figure 3. 

At the following step S5, the unit 40 for recovering the index of 
stored images will recover, in the database, the indexes associated with the 
5 images amongst at least one subset of the stored images. As explained 
previously, this index comprises a content data item. 

The indexes extracted from the database are then saved in the 
image data storage unit 20. 

In accordance with the embodiment chosen and depicted, the 
10 index of any image of the database consists of a histogram of colours relating 
to the overall content of the image. This histogram is denoted: H M (D) where D 
designates any image in the database, and M designates the total number of 
colours available (palette of colours), each histogram H M (D) thus containing M 
elements. 

15 Moreover it should be noted that, in the preferred embodiment of 

the invention, the colour space used is the RGB space, however, any other 
space, for example the HSV space, can also be used. 

As mentioned above, it may be sufficient to extract from the 
database the indexes associated with only a part (a subset) of the stored 

20 images. In fact, this may be the case when the database is organised 
hierarchically, so as to group together the images in the database in different 
classes. Each of the classes is then represented by an index having a different 
semantic value. Thus, by defining a measurement of similarity associated with 
each of these indexes, a number of calculations of similarity less than that 

25 which would have been obtained if the base had not been organised 
hierarchically would be effected. 

At step S6, the similarity calculation unit 60 recovers the indexes 
of the stored images and the index of the example image previously stored in 
the unit 20, and effects a calculation of similarity between the example image 

30 and each of the stored images whose index has been extracted from the 
database. 
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In accordance with the present invention, the calculation of 
similarity (effected at step S6) between the example image and a stored image 
under consideration takes into account, through the index of the example image 
- the calculation of which taking into account the regions of interest (ROI r ) and 
5 the associated data (V r ) - the choice of the user with regard to the strategy to 
be applied for seeking images in the database. 

The similarity calculation method according to the preferred 
embodiment of the invention will be detailed below in relation to Figure 4. 

Once the similarity calculation has been made at step S6, a 
10 sorting and a selection of the stored images which have been compared with 
the example image, according to their degree of similarity with the example 
image, are carried out at step S7. 

For example, only the stored images whose degree of similarity 
calculated is greater than a predefined threshold are kept. Amongst the latter 
15 only a predefined number of images (for example ten) are selected, these 
having the highest degree of similarity. The selected images are then displayed 
on the screen (step S8) in an order of similarity (increasing or decreasing). The 
user then makes his choice. 

As a variant, all the images which have been compared with the 
20 example image are displayed on the screen in an order of similarity. 

In another variant, only the image in the database which has the 
greatest similarity with the example image is displayed. 

The steps S7 (sorting/selection) and S8 (display) are implemented 
by the image selection unit 70, of the image search device 1 depicted in Figure 
25 1. 

The control unit 80 of the device 1 controls the sequencing of the 
steps of the image search method and manages the interoperability between 
the different units constituting said device. 

With reference now to Figure 3, a description will be given of the 
30 method of indexing the example image used at step S4 of Figure 2 previously 
described. 
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Figure 3 depicts a flow diagram divided into three parts 
respectively depicted by Figures 3a, 3b, and 3c. 

Commencing with Figure 3a, the method of indexing the example 
image begins at step S401, in which the data associated with the regions of 
5 interest defined in the example image and stored in the storage unit 20 are 
recovered. 

The example image is designated by the letter "Q", these data are 
designated by ROI(Q) containing the data indicating the shape and location of 
these regions of interest in the image Q, and by V(Q) containing the scalar data 
10 associated with the regions of interest. 

As can be seen, the image Q has an integer number R of regions 
of interest ROh to ROI R , and the same number R of associated scalar data Vi 
to V R . 

At the following step, S403, a counter r is initialised to 1 , and then 
15 at step S405 a data item V r is selected (commencing with since the counter r 
was initialised to 1). 

At the following step, it is determined whether or not the data item 
V r is equal to the predefined lower limit V m j n which is chosen here equal to zero 
(0). Moreover the predefined upper limit V max is chosen so as to be equal to 
20 one(1). 

In the affirmative, an indicator of the region of interest ROI r 
associated with the current data item V r is stored (step S409) in a register SO 
representing all the regions of interest whose associated value is equal to V min . 

In the negative, the following step S41 1 is passed to, in which it is 
25 determined whether or not the data item V r is equal to the predefined upper limit 
Vmax- In the affirmative, the identifier of the region of interest associated with 
the current data item V r is stored (step S413) in a register S1 representing all 
the regions of interest whose associated value is equal to V max . 

In the negative, at step S415 the identifier of the region of interest 
30 associated with the current data item V r is stored in a register Sv representing 
all the regions of interest whose associated value is strictly between V min and 

Vmax- 
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The step S417 which follows is a decision step in which first of all 
the counter r is incremented and next it is determined whether or not the value 
of r is strictly greater than the number R of regions of interest defined in the 
image Q. 

5 In the negative, this means that all the values V r have not yet been 

taken into account. In this case, step S405 is returned to in order to select a 
data item V r corresponding to the new counter value r, and steps S407 to S417 
are recommenced as explained above. 

Returning to step S417, if the value of the counter r is tested to be 
10 strictly greater than the number R of regions of interest, this means that the last 
data item V r selected was V R . Step S419 is then passed to. 

At the end of the steps which have just been described, three 
registers SO, S1, Sv have therefore been obtained, which can be defined as 
sets whose elements are regions of interest of the example image, for which 
15 the data item V r supplied by the user is respectively equal to V min (set SO), 
equal to V max (set S1), and strictly between V min and V max (set Sv). 

The following steps S419 to S443 will make it possible to 
determine, from the respective contents of the registers S1 , S2, Sv, which case 
(i.e. condition) applies amongst a plurality of predefined cases. 
20 In accordance with the invention, the method of indexing the 

example image and therefore the calculation of similarity between a stored 
image and an example image varies according to the predefined case which 
applies. 

Returning to Figure 3a, step S419 is a decision step in which it is 
25 determined whether the registers S1 and Sv are both empty. 

In the affirmative, a variable "CASE" is set to 1 , thus indicating that 
the case which applies is the predefined case number one (hereinafter 
designated by "CASE 1"). This case corresponds to the situation in which all 
the data V r are equal to V min . 
30 In the negative, the following step (S423), according to the same 

principle as at the preceding step, it is tested whether the predefined case 
number 2 applies (designated hereinafter by "CASE 2"), in which all the data V r 
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are equal to V max . In the affirmative, the variable CASE is set to the value 2 
(step S425). 

In the negative it is tested successively whether: 

- case number three applies (S427) (designated hereinafter by "CASE 3"), in 
5 which all the data V r are strictly between zero and one (if SO and S1 are 

both empty); if not 

- case number four applies (S431) (designated hereinafter by "CASE 4") in 
which the set V(Q) of data V r comprises solely data which are equal to V min , 
and others which are equal to V ma x. This condition is tested by verifying that 

10 the sum of the elements contained in SO and S1 (cardinal(SO) + 
cardinal(SI)) is equal to the total number R of data V r (cardinal of V(Q)); if 
not 

- case number five applies (S435) (designated hereinafter by "CASE 5"), in 
which the set V(Q) of data V r comprises solely data which are equal to zero, 

15 and others which are strictly between zero and one (card(SO) + card(Sv) = 
R); if not 

- case number six applies (S439) (designated hereinafter by "CASE 6"), in 
which the set V(Q) of data V r comprises solely data which are equal to V max , 
and others which are strictly between V min and V m ax (card(S1) + card(Sv) = 

20 R); if not, by default, 

- case number seven applies (S443) (designated hereinafter by "CASE 7"), in 
which the set V(Q) of data V r comprises data which are equal to V min , others 
which are equal to V max , and finally others which are strictly between V m j n 
and V max . 

25 Finally, when point A in Figure 3a has been arrived at, the 

applicable case has been determined, and the variable CASE has been set to 
the corresponding value (step S421 or S425 or S429 or S433 or S437 or S441 
or S443). 

The process of indexing the example image then continues to 
30 point A in Figure 3b, by step S445 in which the counter r is reinitialised to 1 . 

At the following step S447, a region of interest ROI r is selected, 
commencing with the first (ROh). It should be noted that the expression 
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"selecting" a region of interest means here considering, in the image plane of 
the example image, the location in terms of 2D coordinates of the region under 
consideration. 

At the following step S449, the histogram of colours H M (ROI r ) 
associated with the region of interest selected is calculated and stored. 

It should be noted once again that M designates the total number 
of colours available (palette of colours), each histogram H M (ROI r ) thus includes 
M elements. 

The following step S451 is a decision step in which first of all the 
counter r is incremented and it is next determined whether or not the value of r 
is strictly greater than the number R of regions of interest defined in the image 
Q. 

In the negative, this means that not all the regions of interest have 
been selected yet. In this case, step S447 is returned to in order to select 
another region of interest corresponding to the new value of the counter r, and 
steps S449 and S451 are recommenced as explained above. 

If the histograms of colours corresponding to all the regions of 
interest have been calculated and saved, in this case the test of step S449 is 
positive, and step S453 is passed to, in which all the histograms are grouped 
together in a vector G R (Q) comprising R components, each of which being the 
histogram of colours previously calculated and corresponding to one of the R 
regions of interest defined in the example image (Q). 

At the following step S455, it is determined whether one of cases 
1 , 2 or 4 applies. For this the variable CASE is tested. 

If the value of this variable is 1 , 2 or 4, then step S457 is passed 
to, in which the index of the example image is defined as being the vector 
G R (Q) calculated at the previous step S453, and the process of indexing the 
example image is terminated. 

Conversely, if the test step S455 is negative, S459 is passed to, in 
which the global histogram of colours H M (Q) of the example image is 
calculated, this histogram comprising one element for each of the M colours C m 
in the palette. Point B in Figure 3b is then arrived at. 
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The indexing process continues to point B in Figure 3c, from 
which step S461 is passed to, which is an initialisation step. At this step, the 
counter is reinitialised to 1. 

In addition, variables Wjj with i, j representing any images between 
5 1 and M (number of colours available) are initialised to 0 if i is different from j, or 
to 1 if i is equal to j. 

As these variables Wy are intended to represent the elements of a 
matrix W with M rows and M columns, where each element of the diagonal is 
associated with one of the colours in the palette, the elements of the diagonal 
10 of this matrix are initialised to 1, the other elements being initialised to zero. 
The role of the matrix W will be explained in the remainder of the description. 

After this initialisation step, step S463 is passed to, in which a 
histogram of colours of a region of interest previously calculated and saved 
(step S449) is selected, commencing with the one corresponding to the region 
15 of interest referenced by ROh. 

At the following step S465, the dominant colours C d of the 
histogram selected are identified, using a conventional method based on a 
comparison with respect to a predetermined threshold of the intensity 
associated with a given colour. 
20 At the following step S467, for each dominant colour C d of the 

current region of interest ROI r , extracted at the previous step, there is added to 
the element w dd of the diagonal of the matrix W corresponding to this dominant 
colour the value of the data item V r associated with the current region of 
interest ROI r (w dd = w dd + V r ). 
25 At the following step S469, it is determined whether all the 

histograms corresponding to the regions of interest of the example image have 
been taken into account. If such is not the case, another histogram of colours 
is selected (S463) and steps S465 to S469 are recommenced. 

When steps S463 to S467 have been performed for all the 
30 histograms, each element of the diagonal of the matrix W has been increased 
by the values of the data V r associated with the regions of interest for which the 
colour associated with the said element is dominant. 




20 



For example, if the colour C12 is dominant solely in the region of 
interest ROh and in the region of interest ROI 3 , at the end of the process the 
value of the element w 12 i2 of the matrix will also be at its initial value (1) 
increased by the values of the data and V 3 associated respectively with the 
5 regions of interest ROh and ROI 3 . 

Returning to Figure 3c, at step S471, each element on the 
diagonal of the matrix is standardised to 1, dividing each of these elements by 
the sum of the elements of the diagonal. 

At the following step S473, a global histogram of colours X(Q) of 
10 the example image (Q) is calculated, where each colour element is weighted by 
the standardised element corresponding to this colour of the matrix W. This is 
effected by calculating the product of the matrix W and the colour histogram 
Hm(Q) of the example image, calculated previously at step S459. 

The following step S475 is a decision step in which it is 
15 determined whether CASE 3 applies. For this the variable CASE is tested. 

If the case which applies is indeed CASE 3, step S477 is passed 
to, in which the index of the example image is defined as being the weighted 
histogram of colours X(Q) calculated at the previous step S473. 

Conversely, if it is not CASE 3 which applies, this means, since 
20 cases 1 , 2 or 4 also do not apply (step S455), that CASE 5, 6 or 7 applies. 

Under these circumstances, at the final step S479, the index of 
the example image is defined as being composed of two elements, on the one 
hand the weighted overall histogram of the image, X(Q), and on the other hand 
the vector G R (Q) previously calculated (S453). 
25 Thus, see below a summary of the different indexes (Index(Q)) 

calculated for the example image (Q) according to the predefined case (CASE) 
which applies: 

o If CASE = 1 , 2 or 4 then Index(Q) = G R (Q) 

with G R (Q) = [H M (ROIi), Hm(ROIr)] 
30 o If CASE = 3 then Index(Q) = X(Q) 

with X(Q) = W.Hm(Q) 
o If CASE = 5, 6 or 7 then Index(Q) = [ X(Q), G R (Q) ] 
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The method of calculating the index of the example image 
depends consequently on the data V r associated with the regions of interest 
defined in the example image. The value of said data, through the predefined 
cases mentioned above, causes an adaptation of the image research strategy 
to the meaning of these values. 

This change in strategy results in a method of calculating the 
similarity which varies according to the predefined case which applies, as will 
be explained below. 

At the end of the step of indexing the example image S4 (Fig. 2) 
which has just been detailed with the help of Figure 3, next comes the step (S6) 
of calculating the similarity between the example image and the images stored 
in the database. 

Following this similarity calculation, the images in the database 
are sorted and selected according to the similarity value calculated for each of 
them (in accordance with step S7, Fig. 2). 

The similarity step and the sorting/selection step will now be 
described in detail in relation to Figure 4. 

Figure 4, composed of Figures 4a, 4b, 4c, depicts a flow diagram 
illustrating the method of calculating the similarity between an example image 
and an image in the database in accordance with the embodiment of the 
invention chosen and depicted. 

It should be stated here that, in accordance with the embodiment 
chosen and depicted, the index of any image D in the database consists of the 
histogram of colours relating to the overall content of the image. This index is 
denoted H M (D), M designating, as before, the number of colours available. 

Commencing with Figure 4a, the method of calculating the 
similarity between the example image Q and the images D stored in the 
database begins with the step S601 in which the index of the example image 
calculated previously is recovered from the storage unit 20. 

Steps S603, S609 and S615 of Figure 4a, and then steps S621, 
S631 of Figure 4b, and finally step S641 of Figure 4c, are decision steps in 
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which it is successively determined whether respectively the predefined cases 
1,2,3, 4, 5 or 6 apply. 

If at one of these steps, except for step S641 , it is determined that 
the corresponding predefined case does not apply, the following decision step 
5 is passed to in order to test whether the following predefined case applies. 

When at step S641 it is determined that CASE 6 does not apply, 
then it is considered that CASE 7 applies since all the cases preceding CASE 6 
were tested negatively previously. In this eventuality, steps S551 to S661 are 
performed in order to apply the appropriate processing to CASE 7, and the 
10 similarity calculation process ends there. 

Returning to Figure 4a, if at step S603 it is determined that CASE 
1 applies, then successively steps S605 and S607 are performed in order to 
apply the appropriate processing. 

To this end, at step S605 there is calculated, for each stored 
15 image D, a similarity measurement designated by SIMi, and obtained by means 
of the following formula: 

SIM, (D) = Max [H M (D) fl H M (ROI s r ° )] (1 ) 

in which: 

H M (D) designates the histogram of colours associated with the 
20 stored image under consideration (index); 

ROI s r ° designates any region of interest in the example image 
belonging to the previously defined set SO, that is to say a region of interest for 
which the associated value V r is equal to V m j n ; 

H M (ROIr°) designates the histogram of colours associated with 
25 this region of interest (obtained previously at the step S449 of indexing the 
example image); 

the operator fl designates the operation of intersection between 
histograms; and 

the function Max takes the largest value obtained by these 

30 intersections. 
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The operation of intersection between histograms is known from 
the state of the art. In order to obtain more details on intersection between 
histograms, reference can be made for example to the article by M. J. Swain 
and D. H. Ballard, entitled "Color Indexing", International Journal of Computer 
5 Vision, 7:1, 1991. 

Returning to Figure 4a, after the similarity calculation SIMi for 
each image D in the database (DB) at step S605, step S607 is passed to, in 
which the images in the database are sorted in increasing order ("Sort +") of the 
similarity values obtained. 
10 Finally, amongst these ordered images, a predefined number 

thereof will be selected, for example ten, from amongst those whose similarity 
value obtained is the lowest. The similarity calculation process is then 
terminated. 

The images finally selected constitute the result of the search and 
15 will be presented (displayed) to the user, so that the latter can make his choice. 

It should be stated here that CASE 1 which has just been 
examined corresponds to the situation in which all the data V r are equal to V min) 
which means that the user is seeking images whose content is not similar to 
that of the regions of interest defined in the example image. 
20 Thus, in this eventuality, the search method according to the 

present invention makes it possible to select, in the database, images whose 
content is the least similar to that of the regions of interest defined in the 
example image. 

At step S609 in Figure 4a, if it is determined that CASE 2 applies, 
25 then steps S611 and S613 are performed successively in order to apply the 
appropriate processing. 

To this end, at step S611 there is calculated for each stored 
image D a measurement of similarity designated SIM 2 , and obtained by means 
of the following formula: 
30 SIM 2 (D) = Max [H M (D) fl H M (ROI s r l )] (2) 

in which: 
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H M (D) designates the histogram of colours (the index) 
associated with the stored image under consideration; 

ROI s r l designates any region of interest in the example image 
belonging to the previously defined set S1 , that is to say a region of interest for 
5 which the associated value V r is V max ; 

H M (ROI* 1 ) designates the histogram of colours associated with 
this region of interest (obtained previously at the step S449 of indexing of the 
example image); 

the operator fl and the function Max have the same meaning as 

10 before (SIMi). 

Returning to Figure 4a, after the similarity calculation SIM 2 for 
each image D of the database (DB) at step S611, step S613 is passed to, in 
which the images in the database are sorted in decreasing order ("Sort -") of 
the similarity values obtained. 

15 Finally, amongst these ordered images, a predefined number 

thereof will be selected, for example ten, from amongst those whose similarity 
value obtained is the greatest. The similarity calculation process is then 
terminated. 

The images finally selected constitute the result of the search and 
20 will be presented (displayed) to the user, so that the latter can make his choice. 

It should be stated here that CASE 2 which has just been 
examined corresponds to the situation in which all the data V r are equal to V max , 
which means that the user is seeking images whose content is the most similar 
to that of the regions of interest of the example image. 
25 Thus in this eventuality the search method according to the 

present invention makes it possible to select from the database images whose 
content is the most similar to that of the regions of interest defined in the 
example image. 

Likewise, at step S615 in Figure 4a, if it is determined that CASE 
30 3 applies, then steps S617 and S619 are successively accomplished in order to 
apply the appropriate processing. 
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To this end at step S617 there is calculated for each image D 
stored in the database (DB) a measurement of similarity designated SIM 3| and 
obtained by means of the following formula: 



H M (D) designates the histogram of colours (the index) 
associated with the stored image under consideration; 

X(Q) is the weighted histogram of colours associated with the 
example image and calculated at step S473 in Figure 3c, during the process of 
1 0 indexing the example image: X(Q) = W • H M (Q) . 

Returning to Figure 4a, after the similarity calculation SIM 3 for 
each image D in the database (DB) at step S617, step S619 is passed to in 
which the images of the database are sorted in decreasing order ("Sort -") of 
the similarity values obtained. 
15 Finally, amongst these ordered images, a predefined number 

thereof will be selected, for example ten, from amongst those whose similarity 
value obtained is the greatest. The similarity calculation process is then 
terminated. 

The images finally selected constitute the result of the search and 
20 will be presented (displayed) to the user, so that the latter can make his choice. 

It should be stated here that CASE 3 which has just been 
examined corresponds to the situation in which all the data V r are strictly 
between V min and V max , which means that the user is seeking images whose 
content is more or less similar to that of the regions of interest of the example 
25 image, depending on whether the value of V r is close to V max or V mini whilst 
taking into consideration the global content of the example image. 

Thus, in this eventuality, the search method according to the 
present invention makes it possible to select from the database images whose 
content satisfies most precisely the search criteria defined for CASE 3, through 
30 the use of the weighted histogram X(Q) of the example image. 



SIM 3 (D) = H M (D)ClX(Q) 



(3) 



5 



in which: 
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At step S621 of Figure 4b now, if it is determined that CASE 4 
applies, then steps S623 to S629 are performed successively in order to apply 
the appropriate processing. 

It should be stated here that CASE 4 corresponds to the situation 
in which there exist data V r associated with regions of interest (ROIs) of the 
example image which are equal to V max . and others which are equal to V min (but 
none lying strictly between V min and V max )- 

At step S623 a similarity measurement according to the method of 
calculating SIM 2 (formula 2) is first of all calculated for each image D stored in 
the database (DB). 

Next, at step S625, the images in the database (DB) are sorted in 
decreasing order ("Sort -") of the similarity values SIM 2 obtained. 

Then, amongst the set DB of images thus ordered in the 
database, a subset denoted DB1 will be selected therefrom, for example thirty, 
from amongst those whose similarity value obtained is the greatest. 

At this first stage, there has therefore been selected a subset DB1 
of images in the database whose content is most similar to the ROIs of the 
example image belonging to the set V1 (ROIs whose associated data item V r is 
V ma x). 

At the following step S627, there is calculated, for each image of 
the subset of images DB1 selected at the previous step, a similarity 
measurement according to the method of calculating SIMi (formula 1). 

Next, at step S629, the images of the subset DB1 of the database 
are sorted in increasing order ("Sort +") of the similarity values SIMi obtained. 

Finally, amongst this subset DB1 of images thus ordered, a 
predefined number thereof is selected, for example ten, from amongst those 
whose similarity measurement SIMi is the lowest. The similarity calculation 
process is then terminated. 

Thus, in CASE 4 which has just been disclosed, the result of the 
search consists of a certain number of images whose content is both the most 
similar to the ROIs of the example image belonging to the set V1 , and the least 




similar to the ROIs belonging to VO (ROIs whose associated data item V r is 

V min ). 

These images finally selected are presented (displayed) to the 
user ordered according to their final degree of similarity, so that he can make 
5 his choice. 

In a similar fashion, at step S631 in Figure 4b, if it is determined 
that CASE 5 applies, then steps S633 to S639 are performed successively in 
order to apply the appropriate processing. 

It should be stated here that CASE 5 corresponds to the situation 
10 in which the set of data V r comprises solely data which are equal to V min , and 
others which are strictly between V min and V max . 

At step S633 there is calculated first of all for each image D stored 
in the database (DB) a measurement of similarity according to the method of 
calculating SIMi (formula 1). 
15 Next, at step S635, the images in the database (DB) are sorted in 

increasing order ("Sort +") of the similarity values SIM 1 obtained. 

Then, amongst the set DB of images thus ordered in the 
database, a subset denoted DB1 will be selected therefrom, for example thirty, 
from amongst those whose similarity value obtained is the lowest. 
20 At this first stage, there has therefore been selected a subset DB1 

of images in the database whose content is the least similar to the ROIs of the 
example image belonging to the set VO (ROIs whose associated data item V r is 
V min ). 

At the following step S637, there is calculated for each image of 
25 the subset of images DB1 selected at the previous step a similarity 
measurement according to the method of calculating SIM 3 (formula 3). 

Next, at step S639, the images of the subset DB1 of the database 
are sorted in decreasing order ("Sort -") of the similarity values SIM 3 obtained. 

Finally, from amongst this subset DB1 of the images thus ordered, 
30 a predefined number thereof are selected, for example ten, from amongst those 
whose similarity measurement SIM 3 is the greatest. The similarity calculation 
process is then terminated. 
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Thus, still in CASE 5 which has just been disclosed, the result of 
the search consists of a certain number of images whose content is in 
accordance with the search criteria defined by the user through the values V r 
associated with the regions of interest of the example image. 

These images finally selected will be presented (displayed) to the 
user ordered according to their final degree of similarity, so that he can make 
his choice. 

At step S641 in Figure 4c now, starting from point A issuing from 
Figure 4b, if it is determined that CASE 6 applies, then steps S643 to S649 are 
accomplished successively in order to apply the appropriate processing. 

It should be stated here that CASE 6 corresponds to the situation 
in which the set of data V r comprises solely data which are equal to V max , and 
others which are strictly between V mjn and V max - CASE 6 is consequently very 
similar to CASE 5. 

At step S643 there is first of all calculated for each image D stored 
in the database (DB) a measurement of similarity according to the method of 
calculating SIM 2 (formula 2). 

Next, at step S645, the images in the database (DB) are sorted in 
decreasing order ("Sort -") of the similarity values SIM 2 obtained. 

Then, amongst the set DB of images thus ordered in the 
database, a subset denoted DB1 will be selected, for example thirty, amongst 
those whose similarity value obtained is the greatest. 

At this first stage, a subset DB1 of images in the database has 
therefore been selected whose content is most similar to the ROIs of the 
example image belonging to the set V1 (ROIs whose associated data item V r is 
V max ). 

At the following step S647, there is calculated for each image of 
the subset of images DB1 selected at the previous step a similarity 
measurement according to the method of calculating SIM 3 (formula 3). 

Next, at step S649, the images of the subset DB1 of the database 
are sorted in decreasing order ("Sort -") of the similarity values SIM 3 obtained. 
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Finally, amongst this subset DB1 of the images thus ordered, a 
predefined number thereof are selected, for example ten, from amongst those 
whose similarity measurement SIM 3 is the greatest. The similarity calculation 
process is then terminated. 
5 Thus, still in CASE 6 which has just been disclosed, the result of 

the search consists of a certain number of images whose content is in 
accordance with the search criteria defined by the user through the values V r 
associated with the regions of interest of the example image. 

These images finally selected will be presented (displayed) to the 
10 user ordered according to their final degree of similarity, so that he can make 
his choice. 

Returning to Figure 4c, if none of the previous cases (CASE 1 to 
6) has been validated, this means that CASE 7 applies, in which the set (V(Q)) 
of data V r comprises data which are equal to zero, others which are equal to 
15 one and finally others which are strictly between zero and one. 

Steps S651 to S661 are then accomplished successively in order 
to apply the appropriate processing. 

At step S651 a similarity measurement according to the method of 
calculating SIM 2 (formula 2) is first of all calculated for each image D stored in 
20 the database (DB). 

Next, at step S653, the images in the database (DB) are sorted 
according to the decreasing order ("Sort -") of the similarity values SIM 2 
obtained. 

Then, amongst the set DB of images thus ordered in the 
25 database, a subset denoted DB1 will be selected therefrom, for example thirty, 
from amongst those whose similarity value obtained is the greatest. 

At this first stage, there has therefore been selected a subset DB1 
of images in the database whose content is the most similar to the ROIs of the 
example image belonging to the set V1 (ROIs whose associated data item V r is 
30 V max ). 
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At the following step S655, there is calculated for each image of 
the subset of images DB1 selected at the previous step a similarity 
measurement according to the method of calculating SllVh (formula 1). 

Next, at step S657, the images of the subset DB1 of the database 
5 are sorted in increasing order ("Sort +") of the similarity values SIMi obtained. 

Next, amongst the set DB1 of images thus ordered in the 
database, a subset denoted DB2 will be selected therefrom, for example ten, 
from amongst those whose similarity value obtained is the lowest. 

At this second stage, there has therefore been selected a subset 
10 DB2 of images in the database whose content is both the most similar to the 
ROIs of the example image belonging to the set V1 (ROIs whose associated 
data item V r is V max ) and the least similar to the ROIs of the example image 
belonging to the set V0 (ROIs whose associated data item V r is V min ). 

At the following step S659, there is calculated for each image of 
15 the subset of images DB2 of the database selected at the previous step a 
similarity measurement according to the method of calculating SIM 3 (formula 3). 

Next, at step S661, the images of the subset DB2 of the database 
are sorted in decreasing order ("Sort -") of the similarity values SIM 3 obtained. 

Finally, amongst this subset DB2 of the images thus ordered, a 
20 predefined number is selected therefrom, for example five, from amongst those 
whose similarity measurement SIM 3 is the greatest. The similarity calculation 
process is then terminated. 

Thus, still in CASE 7 which has just been disclosed, the result of 
the search consists of a certain number of images whose content is in 
25 accordance with the search criteria defined by the user through the values V r 
associated with the regions of interest of the example image. 

These images finally selected will be presented (displayed) to the 
user ordered according to the final degree of similarity, so that he can make his 
choice. 

30 With reference now to Figure 5, a description will be given of a 

computer adapted to implement the image search method according to the 
present invention and described above in relation to Figures 2 to 4. It should 
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be noted that the computer illustrated in Figure 5 is a particular and preferred 
embodiment of the general device described above in relation to Figure 1. 

In this embodiment, the image search method according to the 
invention is implemented in the form of a computer program. This computer 
5 program includes one or more instruction sequences whose execution by said 
computer enables the steps of the image search method according to the 
invention to be implemented. 

In Figure 6, the computer 3, which can typically be a 
microcomputer or a workstation, is connected to different peripherals, for 
10 example a digital camera 310 or any other image acquisition or storage device, 
such as a scanner, supplying information (images, video) to be indexed to the 
computer 3. These images can be stored in the storage means available to the 
computer, such as a hard disk 304. 

The computer 3 also has a communication interface 308 
15 connected to a communication network 309, for example the Internet, able to 
transmit to the computer digital information to be indexed. 

The computer 3 also has data storage means such as a hard disk 
304, or a disk drive 305 for writing data to a diskette 306 and reading these 
data. The computer can also have a compact disc (CD ROM) drive (not shown) 
20 on which the images in the database can be stored, as well as a computer card 
reader (PC card) (not shown). 

According to a preferred embodiment of the invention, the 
executable code of the program for implementing the image search method 
according to the invention is stored in the hard disk 304. 
25 According to a variant embodiment, the executable code of the 

program according to the invention is stored in a read only memory 300 (ROM) 
of the computer. 

According to a second variant embodiment, the executable code 
of the program can be downloaded from the communication network 309 via 
30 the communication interface 308 in order to be stored on the hard disk 304. 

The computer 3 also has a screen 302 for displaying the images 
to be indexed or which have been indexed and serving as a graphical interface 




with the user to enable him notably to define an example image and to define 
regions of interest, using a pointing device (not shown) such as a mouse or 
optical pencil, and a keyboard 303. 

The computer has a central processing unit (CPU) 301, for 
5 example a microprocessor, which controls and directs the execution of the 
instructions of the program of the invention stored in the read only memory 300 
or in the hard disk 304. The central processing unit 301 fulfils the function of 
the control unit 80 described above in relation to Figure 1. 

The computer also has a random access memory 307 (RAM) 
10 containing registers intended to store the variables created and modified during 
the execution of the program, notably the variables mentioned above in relation 
to the description of Figure 5, as can be seen in the enlarged representation of 
the RAM in Figure 6. 

Finally, the computer has a communication bus 311 affording 
15 communication and interoperability between the different aforementioned units 
making up the computer 3. 

Naturally, numerous modifications can be made to the 
embodiments of the invention described above without departing from the 
scope of the invention. 
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